Method of providing street view data of a real estate property

ABSTRACT

Video drive-by data provides a street level view of a neighborhood surrounding a selected geographic location. A video and data server farm incorporates a video storage server that stores video image files containing video drive-by data corresponding to a geographic location, a database server that processes a data query received from a user over the Internet corresponding to a geographic location of interest, and an image processing server. In operation, the database server identifies video image files stored in the video storage server that correspond to the geographic location of interest contained in the data query and transfers the video image files over a pre-processing network to the image processing server. The image processing server converts the video drive-by data to post-processed video data corresponding to a desired image format and transfers the post-processed video data via a post-processing network to the Internet in response to the query.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority under 35U.S.C. §120 from, nonprovisional U.S. patent application Ser. No.13/481,912 entitled “Three Dimensional Image Data Models,” filed on May28, 2012. Application Ser. No. 13/481,912, in turn, is a continuationof, and claims priority under 35 U.S.C. §120 from, nonprovisional U.S.patent application Ser. No. 12/036,197 entitled “Methods and Apparatusfor Generating Three-Dimensional Image Data Models,” filed on Feb. 22,2008, now U.S. Pat. No. 8,207,964. Application Ser. No. 12/036,197, inturn, is a continuation-in-part of, and claims priority under 35 U.S.C.§120 from, nonprovisional U.S. patent application Ser. No. 11/216,465entitled “Apparatus and Method for Producing Video Drive-By DataCorresponding to a Geographic Location,” filed on Aug. 31, 2005, nowU.S. Pat. No. 7,389,181. Application Ser. No. 11/216,465, in turn,claims priority under 35 U.S.C. §120 from provisional U.S. patentapplication Ser. No. 60/605,498 entitled “Apparatus and Method forProducing Video Drive-By Data Corresponding to a Geographic Location,”filed on Aug. 31, 2004. The subject matter of each of the foregoingdocuments is incorporated herein by reference.

TECHNICAL FIELD

The present application relates in general to systems and methods ofgenerating image data of a geographic location of interest.Specifically, the invention relates to generating video drive-by datacorresponding to a selected geographic location. The video drive-by dataprovides a street level view of a neighborhood surrounding the selectedgeographic location, and can be further combined with other data sourcesin a variety of applications. The invention also relates to generatingdata descriptive of a continuum of images, such as those comprising ageographic setting.

BACKGROUND INFORMATION

Companies and individuals looking for real estate to rent, buy, insureor finance have historically had to physically travel to the property inorder to visually verify the condition of the property. With the adventof the Internet, it has become possible for individuals to viewphotographs of properties on-line in the comfort of their home oroffice. One such example is the MLS (Multiple Listing Service) database,which can be accessed by the public through Internet sites such aswww.realtor.com. The MLS database typically includes one or morephotographs and additional data regarding the property such as thenumber of bedrooms and the square footage. More advanced systems mayinclude short video clips constituting virtual tours of the property inquestion. While such systems may also show views of a property from thestreet, they do not give a user a sense of the neighborhood in which theproperty is located. Accordingly, in order to determine if the areasurrounding the property of interest is desirable, a user may often berequired to physically travel to the property in order to view the stateof the neighborhood.

In order to overcome the deficiencies of present systems of the typedescribed above, it would be desirable to provide a system that wouldenable a user to view the surrounding neighborhood as well as theproperty itself. Such a system would enable the creation of a completeonline database of street-level video data, hereinafter referred to as“video drive-by” data, which can be made available to end users via theInternet. Preferably, the video data would be encoded with GPScoordinates as well as conventional street addresses and map coordinatesto provide instant street level video of any location upon request. Forexample, such a system would be extremely useful if a user could querythe system by submitting GPS coordinates, street addresses or by“clicking” on a map overlay showing streets in the desired neighborhood.

The ability of a system to provide data associated with theneighborhood, as well as a particular property in question, would beextremely desirable in a number of government and commercialapplications. Aiding in the process of appraising a property, forexample, is just one application in which such a system can providebenefits. Mortgage lenders currently purchase several types of appraisalproducts as part of their financing decision process. The lowest costappraisal is an automated valuation model (AVM), whereby county parceldata of subject property can be compared to the county parcel data of alist of nearby properties (called “comps” which is short forcomparables) with a similar lot size, square footage, and amenities(such as waterfront access). A comp list typically includes average persquare foot calculations of the similar properties that enables thecreation of approximate value of the subject property. The next lowestcost appraisal is a Desk Review whereby an appraiser reviews the AVMdata on behalf of the mortgage company and evaluates the relevance ofthe data presented in the AVM in conjunction with what they personallyknow regarding the area the subject is located in. A separate class offield appraisals includes both a Drive By Appraisal and a Full InteriorAppraisal. A Drive By Appraisal is where an appraiser drives by andshoots a photograph from the street of a subject property and it'scomps. These photographs are linked to county parcel data and AVMcalculations to provide a more accurate report to the mortgage company.Finally, for a Full Interior Appraisal which is the most expensive andconsidered most accurate appraisal, an appraiser will tour the interiorof a property taking numerous photographs, measuring and sketching afloor plan as well as notating the general quality of construction andupkeep, this data is then presented as a multi property report whichincludes tabular data of subject and comp data with comp photos.

In the case of three of four appraisal types, however, a mortgage bankermust wait several days before receiving the information from anappraiser. If a system were available to allow quick access to allproperties within a neighborhood, a mortgage banker could obtainphotographs of a subject property and its comps and confirm that thecomparables as selected by the AVM represents properties that have thesame visual appearance of the subject property, thereby providing fasterloan commitments and closings while at the same time lowering costs.Similarly, instant Drive By Appraisals could be provided therebyeliminating the costs and delays associated with an appraiser driving tomanually photograph a subject property and its comps.

Images of geographic areas created on a local basis have been created innumerous formats. Photographs, movie cameras, video camera recorders,and more recently digital recorders have all been utilized to capturemoving images of a geographic parcel. Photographs can be useful becausethey are easily associated with a particular real estate parcel;however, they are limited in the scope of the subject matter that theycan include. In addition, photographs must be manually acquired anddocketed in order to be associated with a property and subsequentlyretrieved. Panoramas can increase the scope of the subject matter to apoint, but are limited by a point of capture of the panoramic views.

Movie cameras, video recorders and digital recorders and other “motionpictures” provide for increased scope of image capture. However, it issometimes difficult to locate and view a particular portion of interestof images captured as motion pictures. In addition, correlation ofparticular portions of a motion picture with a particular real estateparcel can be difficult if the motion picture includes more than onereal estate parcel. For example, it is relatively difficult to locateand view a particular real estate parcel in the context of itsneighborhood setting, or particular aspects of its setting. Aerialimages, such as satellite pictures of geographic areas have also beencaptured, and specific parcels of land or landmarks can be located onthe aerial images.

In addition, methods and apparatus have been known for generating pointclouds and generating representations of particular objects fromprocessing of the point clouds.

However, prior to the present invention, there has not been a mechanismfor generating a continuum of object representations based upon pointcloud data. Nor has there been a mechanism for accurately correlatingground level images with substantial portions of an aerial image in aconsistent and orderly format that allows for the identification of aparticular parcel of land and provide both aerial and ground level viewsof the parcel, as well as a surrounding area of the parcel. Similarly,prior to the present invention, there has not been a method forcorrelating geopolitical indicators, such as property lot lines, or taxmap parcels with aerial images and ground level video images.

Substantial portions of ground level images have not been correlatedwith aerial images or with geopolitical indicators, in part, becausethere has not been any image vehicle capable to providing a format ofhigh quality and accurate representations of street level images capableof being matched to macro level image data.

SUMMARY

A system and method of providing video drive-by data is provided toenable a street level view of a neighborhood surrounding a selectedgeographic location. The video drive-by data can be further combinedwith other data sources related to the geographic location.

The invention is based, in part, on providing: 1) accurate differentialGPS data; 2) post processing of geo positioning signals to smooth curvesin motion (splines); 3) highly accurate camera position and video frameposition analysis processing to provide a calculation of the exactposition of each video frame; 4) parcel data processing that analysesvector line data that is geo-coded with latitude and longitude values;5) digital image photos processed with image superimposition algorithms;and 6) a database that includes video image files, parcel latitude andlongitude data, and positioning data that is indexed to the video imagefiles. With these components, the invention enables the access to videoimages of any desired geographic location and its surroundingneighborhood, while relating such image data to other property relateddata such as property lines, landmarks, etc.

In a preferred embodiment, a system is provided that includes a videoand data server farm. The video and data server farm includes at leastone video storage server that stores video image files containing videodrive-by data that corresponds to a geographic location, a databaseserver that processes a data query received from a user over theInternet that corresponds to a geographic location of interest, and animage server. In operation, the database server identifies video imagefiles stored in the video storage server that correspond to thegeographic location of interest contained in the data query, andtransfers the video image files over a pre-processing network to theimage processing server. The image processing server converts the videodrive-by data to post-processed video data corresponding to a desiredimage format, and transfers the post-processed video data via apost-processing network to the Internet in response to the query. Alanding zone server is preferably provided to receive the video drive-bydata from a portable memory device and permit the viewing and analysisof the video drive-by data prior to storage in the video storage server.Still further, a map server is preferably provided to present a staticimage of an overhead view of the geographic location of interest.

The video drive-by data is preferably captured by a video capture systemthat includes a camera array that generates video image data of aneighborhood corresponding to a geographic location, a positioning unitthat produces positioning data corresponding to the location of thecamera array, and a processing unit that processes the video image dataand positioning data to produce the video drive-by data. The processingunit stores the video drive-by data on the portable memory device. In apreferred embodiment, the camera array includes a plurality of cameraunits arranged to provide a 360 degree view of the neighborhood.Further, the processing unit preferably provides time stamp data andfile pointer data that permits synchronization of video images toprecise locations of individual frames.

The above-described system can further be provided with a plurality ofprocessing means to add additional features or permit utilization of thesystem for specific applications. For example, processing means may beincorporated to provide a variety of geo-coded data layers inconjunction with the video drive-by data. Similarly, processing meanscan be provided for: creating geo-coded text, image and vector datawhich is superimposed onto the post-processed video data that istransferred to the Internet; linking latitude and longitude data withvideo drive-by data; calculating a camera position of designatedindividual frames of video drive-by data using position data pointscaptured before and after a designated frame position; for examiningchanges of relative position in the horizontal position of designatedlandmarks in a sequence of video image frames to create fixed landmarkdata points; examining images that match common image patterns to enablethe creation of an accurate vector map of roads; providing OCRprocessing to generate a database of recognized text that is geo-codedwith latitude and longitude coordinates based on camera orientation andrelative position; and selecting video drive-by data or a given streetaddress by calculating a centroid of a parcel and determining cameraorientation and position to enable highly accurate delivery of videothat is pointed toward the centroid of a parcel.

Still further embodiments can include: processing means by which asequence of video frames that are oriented toward a designated parceland it's adjacent parcels can be stitched together to form a longhorizontal image of a street which can be created with superimposedproperty county parcel data placed above the centroid of the parcelcentered above the property; processing means for providingauthenticated annotation of information or comments to a photo and orvideo; processing means for uploading a digital image photograph to thevideo and data server farm and comparing the uploaded photo to a libraryof images captured by the video capture system; and/or processing forcomparing digital image photos of building structures to generate a setof numeric values representing line segments of sharp horizontal,vertical and angular lines that are stored as vectors each with commonratios of distances from each other, wherein the list of vector linesegments can be searched with a high-speed database query to generate alist of possible matches of a subject property against a large databaseof properties with vector line data on file.

The present invention can be utilized in a variety of applications. Forexample, a system in accordance with the invention can offer invaluableassistance to local, county or state road and highway departments inplanning new projects or tracking existing placements of signage,traffic signals, utility boxes, and other facilities and equipmentlocations. Administrators can preview and familiarize themselves with atargeted location in conjunction with assessing needs or assigning workdetails.

New layers of routing assistance can also be provided to emergencyvehicle dispatching systems by including roadside views to allow rapidand accurate targeting of a destination. Tied in directly to county GISmapping, the present invention can offer street level video into theneighborhood and directly to the street address, offering photographs ofany property transmitted via wireless Internet to help the fire, policeor rescue driver find his or her way.

With the capability to fully integrate, refine and unify existing countyproperty records and GIS mapping resources, the system can enhanceonline presentations with parcel line enhanced ortho imagery, propertyimages and now with video. Further, with the system, unification ofproperty records and county mapping can evolve to tie in to one-clickaccess to street level video segments allowing a user to virtually drivedown any street, while highlighting any property and its surroundingneighborhood with parcel lines drawn on the photograph.

Still further, local development agencies wanting to attract businessand industrial relocations to their communities can insure an edge overcompeting community bids by offering one-click access to visually tourplanned growth areas. Using integrated aerial mapping, property data anddata generated by the system, users can scout available developmentareas, accessing integrated mapping layers to identify a developmentarea's proximity to rail, highways, air and port facilities, while alsoincluding street level video footage of local amenities, residentialneighborhoods, schools, recreation and other visual explorations.

Similarly, tourism boards at any level can promote travel and recreationto a broader audience by integrating visuals and streaming video inpresenting points of interest, accommodations, and dining as integratedlayers to the local, county or state map that are unified to promotionalcontent. The video drive-by data generated by the system can offercompelling video segments highlighting parks, beaches, hunting, fishing,arena sports, museums and places of historical interest; while, alsoproviding directions, drive maps, and links to information detailingplaces to stay, restaurants, rental agencies and more.

In non-governmental applications, the system can be utilized tocommunicate the value and desirability of a new subdivision or locationto potential builders and residents. For example, a property developer'sweb site can be switched on to enable parcel line enhanced satellitephotography with access to street level video data of the newly pavedroads of the development. Web site visitors would have the freedom oftaking a virtual tour of any selected parcel of property, looking aroundto note the scenery, tree line, and proximity to neighboring lots,thereby providing a true feel for the land within its naturalsurroundings. Data and content can be integrated to any parcel with aready feedback mechanism allowing viewers to inquire further or requestan appointment.

Similarly, property appraisers can gain new efficiencies at reducedcosts by utilizing a video drive-by data archive of streaming streetlevel video to select and capture freeze-frame video images of homes forsale needing an exterior photo. The system's unification of propertydata, MLS records, mapping with the video resources allows any propertyto be quickly selected, analyzed, and compared to equivalent propertylistings, generating reports with active links to both data and visualsthat can be emailed at a moments notice.

The present invention also relates to methods and apparatus ofgenerating image data. More specifically, the present invention alsorelates to generating data descriptive of a continuum of threedimensional images, such as those comprising a geographic landscape.Accordingly, the present invention provides methods and apparatus forgenerating a continuum of three dimensional image data. In someembodiments, the continuum of three dimensional image data provides astreet level representation of geographic locations. Two or more sets ofimage data are captured of a subject area, wherein each of the two ormore images are captured from disparate points on a continuum

In another embodiment, methods and apparatus are provided for generatinga three dimensional output that includes a continuum of image datasprayed over three-dimensional models. The three-dimensional models canbe representative of features captured by the image data, wherein imagedata can be captured at multiple disparate points along anothercontinuum. The user interface can also include multiple modalities ofimage data and statistical analysis of the image data.

Yet other embodiments provide methods and apparatus for generating acontinuum of image data. The continuum of image data provides a twodimensional ribbon of street level views of geographic areas. Theribbons are created by capturing two or more images of a subject,wherein each of the two or more images are captured from disparatepoints on a continuum. For example, each of the two or more images canbe captured from a different position on a street as a vehicle drivesdown the street. Generally, the images will be captured from anorthogonal view of the subject.

Portions of the two or more images are then aligned in a dimensionconsistent with the continuum, wherein, according to our example thecontinuum includes the path of the vehicle. Therefore, the images wouldbe aligned in a dimension consistent with the path of the vehicle. Acomposite image of the subject is generated from the aligned portions ofthe two or more images.

In some embodiments, positional data is recorded that is descriptive ofa respective location of each of the two or more images. The positionaldata can include, for example, latitude and longitude coordinates andcan be used to associate the composite image with a particular portionof the subject. It is also within the scope of the invention to recordthe altitude of a camera used to capture the image data and therebyapproximating the altitude of the subject matter of the image data.

In some embodiments, a camera will be maintained approximatelyorthogonal to the subject captured and therefore, those embodiments willmaintain the camera at an angle about between 75° and 105° in relationto the subject matter.

In another aspect, some embodiments can include various overlays ofinformation on top of the continuum of image data. Overlays can includeone or more of: metadata, data related to a composite image, andgeospatial information.

Other advantages, features and applications of the invention will becomeapparent to those skilled in the art from the following detaileddescription of the preferred embodiments of the invention and theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described in detail with reference to certainpreferred embodiments thereof and the accompanying drawings, wherein:

FIG. 1 is a schematic diagram of a Video and Data Server Farm inaccordance with a preferred embodiment of the invention;

FIG. 2 is a schematic block diagram of a Video Capture System thatsupplies video data and corresponding synchronous GPS data to the Videoand Data Server Farm illustrated in FIG. 1;

FIG. 3 is a process flow diagram illustrating the process of capturingthe video data and synchronous GPS data utilized by the Video CaptureSystem illustrated in FIG. 2;

FIG. 4 is a process flow diagram illustrating an efficient process inaccordance with the present invention by which numerous field driverswith Video Capture Systems of the type illustrated in FIG. 2 can becontinually supplied with hard disk drives to capture data;

FIG. 5 illustrates a web browser view with an address bar pointing to aweb page that shows both an ortho image view of a mapserver and videopane in accordance with the invention;

FIG. 6 is a process flow diagram for superimposing geo-coded text dataover video captured by the Video Capture System of the type illustratedin FIG. 2;

FIG. 7 is a data flow diagram that illustrates the process by which aninstant drive-by appraisal or Photo AVM (Automated Valuation Model) canbe created using a Video drive-by database that contains geo-codedvideo;

FIG. 8 illustrates a top perspective of a Video Capture System as itdrives past a structure;

FIG. 9 illustrates a frame position interpolation in accordance with thepresent invention;

FIG. 10 shows a set of four illustrations which represent a single frameor portion of a window within an Internet Web Page in accordance withthe invention;

FIG. 11 illustrates a process by which images captured by a left facingand a left front facing camera in a Video Capture System can calculateand generate superimposed property;

FIG. 12 illustrates a process to extract and create a point of interestdatabase that is encoded with GPS data;

FIG. 13 illustrates two ortho views of a Video Capture System as itdrives in a direction of travel along a path past a selected parcelwhere one of six possible cameras may be viewing a subject property;

FIG. 14 illustrates a process by which an authenticated end user cansubmit data regarding a specific location using Video drive-by data as ageo-coding reference tool;

FIG. 15 illustrates a process and visual references as to thecalculations required to extract a photograph from a street number inaccordance with the invention;

FIG. 16 illustrates a novel method of finding a house from a photographbased on the invention;

FIG. 17 illustrates one novel method of identifying a home from a largelibrary of home photos with a high speed process in accordance with theinvention;

FIG. 18 illustrates a continuum from which image data sets capture asubject;

FIG. 19 illustrates multiple images captured from disparate points alonga continuum;

FIG. 20 illustrates multiple image data sets that overlap subjectmatter;

FIG. 21 illustrates a three-dimensional model generated form the imagedata sets;

FIG. 22 illustrates a user interface generated from the image data setsand including multiple modalities of image data;

FIG. 23 illustrates an exemplary user interface for traversing multiplemodalities of image data;

FIG. 24 illustrates an exemplary composite of image data set portions;

FIG. 25 illustrates apparatus what may be used to implement the presentinvention;

FIG. 26 illustrates basic geographic images captured from a camera atdisparate points along a continuum and aligned with post processing;

FIG. 27 illustrates an alignment of portions of two or more imagescaptured from disparate points along a continuum to form a compositeimage;

FIG. 28 illustrates multiple images captured from disparate points alonga continuum and specific vertical slices that can be selected to createa composite image;

FIG. 29 illustrates a block diagram of exemplary lens distortion factorsand a preferred area of lens capture that can be selected in someembodiments of the present invention;

FIG. 30 illustrates disparate points on a continuum with an orientationvector associated with each respective point;

FIG. 31 illustrates a canyon formed by two continuums of image dataaccording to the present invention; and

FIG. 32 illustrates apparatus that can be used to implement someembodiments of the present invention.

DETAILED DESCRIPTION

A Video & Data Server Farm 0100 in accordance with a preferredembodiment of the invention is illustrated in FIG. 1. The Video & DataServer Farm 0100 utilizes a set of servers to provide digital storage ofvideo drive-by data, and processes the video drive-by data for deliveryto an end user. The video drive-by data preferably includes video dataand Global Positioning System (GPS) data or “geo position data”generated by a Video Capture System 0201 illustrated in FIG. 2.Specifically, video drive-by data is preferably stored on an externalhard disk drive (HDD) 0250 of the Video Capture System 0201, and is thencopied from the external HDD 0250 to a landing zone server 0130 of theVideo & Data Server Farm 0100 for quality analysis and viewing viaindustry standard commercially available MPEG2 viewing software and fileeditors. The Video drive-by data, including the video data and GPS data,is transferred via a load network 0142 to be stored as video image filesin video storage servers 0145 of the Video & Data Server Farm 0110.

In operation, a user logs onto the Internet 0150 via an end userinterface 0102 to access the video image files containing the convertedvideo drive-by data via a data query that includes data related to aspecific geographic location of interest. The data query received by aweb server 0105 and carried over a control network 0135 to a databaseserver 0120 and a map server 0125, which preferably calculates andpresents a static image (for example a street map image) of an overheadview of the geographic location of interest. The database server 0120identifies the appropriate video image files corresponding to thelocation of interest from the video storage servers 0145 for transferover a pre-processing network 0140 to image processing servers 0115. Theimage processing servers 0115 convert the original video drive-by datato one of many potential new image formats (depending on the particularapplication) which constitute Post-processed Video Data (PPVD). The PPVDis then transferred to a video server 0110 over a post-processingnetwork 0141, which then forwards the PPVD data to the Internet 0150 inresponse to the query. In addition, if so desired, the video server 0110can also concurrently transfer the PPVD back to the video storageservers 0145, database servers 0120 and map server 0125 for futureaccess from a video cache.

FIG. 2 is a schematic block diagram of the components in the VideoCapture System 0201. In a preferred embodiment, a camera unit 0205preferably containing six video cameras 0210 in a hexagonal array isprovided in a camera housing assembly 0215. The camera housing assembly0215 is mounted with differing configurations on top of a vehicle (notshown) preferably using commercially available roof racks and straps.The electronics for converting and storing the video data out of thecamera 205 are preferably provided in a passenger cab 0220 of thevehicle. The Video Capture System 0201 is preferably powered from thevehicles electrical system, although a separate power source can beprovided if so desired. The camera 0205 feeds analog video signals 0227along six cables in a cable harness 0225 which is fed through a smallopening in a window of the vehicle. The analog video signals 0227 arefed into six analog-to-digital (A/D) converters 0230, which convert theanalog video signals 0227 to compressed digital video data. Thecompressed digital video data is then sent on data cables 0232 into aseven port USB hub 0235. The six streams of digital video data are fedvia the hub 0235 into a laptop computer 0245, which preferably writesthe digital video data onto a portable storage device, for example theexternal HDD 0250, simultaneously with the data from a GPS receiver0240. Accordingly, the data is stored with corresponding GPSsynchronized coordinates in the external HDD 0250 as the Video drive-bydata.

FIG. 3 is the block diagram illustrating the process of capturing thevideo data and synchronous GPS data to create the video drive-by data. Aset of programs is automatically launched when power is applied to thelaptop computer 0245. The GPS capture program 0305 processes the datatransmitted from the GPS receiver 0240 and writes 0307 this as timestamped data to the laptop computer's RAM 0330 and to a file on theexternal HDD 0250. The time stamp of the data file written contains thelaptop computer's internal system clock time data as well as the GPSclock data. A driver viewer application 0310 is initialized to enablethe driver of the vehicle to confirm that the GPS receiver 0240 isworking, and—as the video data is captured—the driver can preferablyview segments of the video data on the display of the laptop computer0245 to confirm the system is working properly. The driver viewerapplication 0310 shows a map of roads with color codes representingareas that have already been surveyed in one color and streets that needto surveyed in a second color.

Video capture daemons 0315 are started to process the video drive-bydata arriving via the USB hub to then write 0317 the six data files tothe external HDD 0250 with the laptop computer's 0245 system time clocktime stamp data and file pointer data added to the video file and a GPSlog file. In addition, the video capture daemons writes 0317 the filepointer data to the laptop computer's RAM 0330 for use by a master indexfile. The master index file daemon 0340 is started which reads the GPSdata, the six video file pointers values and then writes 0345 the masterindex file data to the external HDD 0250. The entire set of data is thencopied to the landing zone server 0130 when the external HDD drives aresent to the Video & Data Server Farm 0100 for upload and processing aswill be described in greater detail below.

FIG. 4 is a process flow diagram illustrating a preferred efficientprocess by which numerous field drivers with video capture systems canbe continually supplied with hard disk drives to capture data. Thefigure is comprised of three columns 0405, 0410 and 0420 separated bydashed lines. The left column 0405 entitled “Video & Data Server Farm”represents the location of the Video & Data Server Farm 0100 where alldata is stored and subsequently delivered to the Internet 0150. Themiddle column 0410 entitled “In Transit Shipping” represents themovement of data in external hard disk drives 0250 via commercialshipping companies. The right column 0420 entitled “Video Capture SystemField Driver Residence” represents one of many locations where driversin the field handle external hard disk drives for receiving, installingand shipping.

In the illustrated example, time movement is indicated along thevertical axis in a downward direction with the start of the process atthe top indicated as Week 1. For example, the end of Week 1 isrepresented by a horizontal line 0419. During Week 2, represented as thetime between the lines 0419 and 0429, a shipment 0443 of blank externalhard disk drives 0445 is moved from Video & Data Server Farm 0405 to theVideo Capture System Field Driver Residence 0420. The angle of the linein the downward direction to the right illustrates the time delay ofseveral days as the external hard disk drives 0445 are physically movedby a commercial shipping company or private shipping means specificallyset up for operation of the system.

The process starts with an initial shipment 0430 of a Video CaptureSystem through In Transit Shipping process path 0432 to individualdrivers in the field for installation in their respective vehicles. Ablank external HDD is then shipped in process path 0433 to theindividual drivers in their respective Field Driver Residence forinstallation into the Video Capture System. The external HDD 0250 isillustrated as several unique external hard disk drives as they changestate and use as they move through various stages of processing. Forexample, the first external hard disk drive labeled as HDD1 0440 isshipped blank to the Field Driver Residence where it is installed.During Week 2, a driver drives along selected roads capturing video dataand GPS data which is then written onto the HDD1 as illustrated with thewavy line 0445. During Week 2, a second blank external hard disk driveHDD2 0492 is shipped 0443 to the field driver where it sits in inventorywaiting for use. At the end of Week 2, the driver ships 0450 the firstfull HDD1 0491 to the Video & Data Server Farm for processing. At thebeginning of Week 3 0429, the first HDD1 0491 is now re-labeled asHDD1-4 in the landing zone upload process 0496 carried out at the Video& Data Server Farm.

In the landing zone upload process 0496, the data is copied 0465 to thelanding zone server and a GPS.LOG and a Master Index File data is loadedto the map server. An operator is given an opportunity to review thedata using video file viewing software to provide a visual qualityreview test process 0470. Once the quality review process 0470 iscompleted, the operator then erases 0475 the data from the external HDDand places the external HDD drive on the inventory shelf 0480 labeled asHDD4.

At the beginning of Week 3, the field driver installs HDD2 0492 into thevideo capture system and starts driving only a survey route to capturevideo data which is then stored 0455. Later in Week 3, the field driverreceives a blank HDD3 0493 which is stored at the field driver residenceuntil it is needed at the beginning of Week 4 0439. The driver completesthe capture process at the end of Week 3 0439 and ships 0463 HDD2 0492to Video & Data Server Farm 0405, at which point the landing zone uploadprocess is repeated 0498. The above-described process provides a simpleyet efficient method of moving large amounts of data from the fielddriver residence back to the Video & Data Server Farm.

The processes for accessing and viewing data will now be described ingreater detail. FIG. 5, for example, illustrates a web browser 0505 viewwith an address bar 0510 pointing to a web page that shows both an orthoimage view 0515 of a mapserver and video pane 0540. In the ortho imageview 0515, drive line layers 0520 are shown representing the GPScoordinates as a map layer on top of parcel lines 0525 with an end usersmouse cursor 0530 pointing to the selected area of video that the userwants to see in the video pane 0540. In this way, a user can select aparcel 0535, indicated with the double lines, and then the system canshow a video view of the property 0545 in the video pane 0540. With thistype of interface, a user can simultaneously view both conventionalstreet map representations as well as the street level Video drive-bydata generated by the system.

FIG. 6 is a process flow diagram of the processes implemented with theimage processing servers 0115 required to superimpose geo-coded textdata over video captured by the Video Capture System. For a given parcelor street address, a video program 0610 determines available videosegments that contain images where the property is in view. This isdetermined by using an open source programming tool GDAL which completesa range calculation from the latitude and longitude positions encoded inthe video segments to the latitude and longitude positions of thesubject property. The determination of if a parcel is in view is basedon a variable distance that is initially set at approximately 300 feetfrom camera position to the nearest line segment of a parcel. Once asegment of video is selected that contains video frames within 300 feet,an extractor program 0620 (which is based on an open source programmingtool MENCODER) converts the compressed MPEG2 data into lossless TIFFimage files with each image frame file encoded for camera orientationand interpolated location.

For the target parcel, a calculation using GDAL is made in a cameracentroid position program 0630 to determine the exact latitude andlongitude of a centroid of the parcel, which generally approximates thelocation of the structure. Given the centroid of the parcel, anothersimple geometry calculation is made as to the orientation and locationof the camera view that is pointed toward the centroid of the parcel.The location of the centroid of the parcel and the distance to theparcel is then calculated for the frame. A render program 0640, whichruns on the image processing servers 0115, is then provided (in thisexample implemented using open source programming tool suite OpenGL andGraphicsMagick) to calculate a location within the TIFF image file inwhich text or an icon can be placed based on the centroid. This processis then repeated in each of the TIFF images which are then stored ineach of the image files. A recompositing program 0650 (in this exampleimplemented using an open source programming tool FFMPEG) then assembleseach of the TIFF images together in time sequence to create a compressedvideo file. Finally, the video in MPEG2 format with the newly added textand icons added is streamed 0660 out to an end user using an open sourceApache web server.

As stated above, one potential application for the above-describedsystem is in the field of real estate appraisal. FIG. 7 is a data flowdiagram that illustrates the process by which an instant drive-byappraisal or photo AVM (Automated Valuation Model) can be created usinga video drive-by database that contains geo-coded video. The diagramrepresents four discrete locations in three columns: the upper leftcolumn represents a Video & Data Server Farm 0701; the center columnrepresents the Internet 0702; the right column represents an end userappraiser 0703; and the lower left column represents a lender 0704. Inoperation, data is transferred from end user appraiser 0703 to and fromthe Video & Data Server Farm 0701 and finally to the Lender 0704 via theInternet 0702.

The process starts, for example, with the end user appraiser 0703 entersa street address 0710 for a subject property on a web form. The streetaddress data is then sent 0715 to the server hosting one of manycommercially available AVM or Comparative Market Analysis Applications(CMAA) 0720, which does a database search and selects a list ofproperties that are similar in price, size and nearby location to thesubject property. The list of comparables or comps is sent 0725 back tothe end user appraiser 0703 who reviews and analyses various data fieldsfor the comps (such as price, sale date, bedrooms, square feet,structure, lot size, etc.) from the full list of comps 0730. Theselected list of comps 0735 is sent back to the Video & Data Server Farm0701 where a select video program 0740 transfers video files from theimage processing servers 0115 to a video-to-digital photo extractorprogram 0745 (implemented in the current example using open sourceprogramming tools MENCODER and DNoise3D). Digital photos of the subjectproperty and selected comps are then sent 0750 to the end user appraiser0703, who is presented with a web form 0760 which contains a photo ofthe subject property 0762 and a set of photos of comps (0764 thru 0768).

In the above illustrated example, the end user appraiser 0703 reviewsthe picture of the first comp 0764 and sees that it is approximately thesame size and appearance and then clicks on the first check box 0784 toaccept it as relevant and appropriate. In the case of the third comp0766 that is presented, it is too small and it is not selected 0786. Thesecond 0765 and fourth comps 0767 are selected 0785, 0787 and finallythe fifth comp 0768 is not selected 0788 as being too large and not ofappropriate visual quality as the subject. The selection list is thensent back 0770 to the lender 0704 as the appraisal selected relevantcomp properties report, which is then reviewed 0775 by the lendingofficer to assist in their financial decision process.

FIG. 8 illustrates a top view perspective of a video capture system inaccordance with a further embodiment of the invention as it drives pasta structure. The line 0800 represents the track of the Video CaptureSystem with one-second GPS time stamp intervals 0801, 0802. Along thepath from 0801 and 0805 there are 30 notch marks with one represented as0805; each notch mark represents 1 of 30 video frames captured eachsecond by a right facing video camera of the camera array. At frame 50810, the right camera field of view 0820 has several easily identifiedhigh contrast landmarks in sight. The term “landmark” represents anyfixed object, coloration or structure that is easily identified in adigital image photograph, this can include (but is not limited to) lightpole, mail box, edge of driveway, corner of building, roofline, frontdoor, tree, bush, pond edge, awning, centerline on road, public sign,etc. The landmarks are identified and labeled in the video frames withcommercially available and open source programs that provide edgedetection analysis of digital image photos. The view of the left edge ofa landmark tree trunk 0830 and its leaf-shading boundary 0831 ishighlighted with an arrow 0835. The horizontal position of the treewithin the field of view is indicated in line 0825. In addition, thecamera has several other landmarks a window frame 0836 and a door 0837that have virtual lines 0826, 0827 drawn from the camera to thelandmark. These virtual lines provide a point reference in thehorizontal dimension with a video frame as to the angle by which thislandmark is oriented from the camera. As the Video Camera system travelsalong the road to another point in Frame 29 0815, which is one frameshort of the second GPS time stamp 0802, a second set of landmark viewsvirtual lines 0845, 0846 and 0848 within the right camera field of view0821 are available for simple geometry calculations. The horizontallocation of the tree trunk landmark 0835 is now in the extreme rightmost position of the video frame of 0815 camera position.

Using simple interpolation techniques, the position of each framebetween the GPS one-second interval points can be calculated with timeand speed between each. The assumptions are that no radical changes inspeed or direction occur within the one second interval. Using simplegeometry the change of horizontal location of the camera from frameposition 5 0810 to frame position 29 0815 pointing to a fixed point 0835enables a calculation of the distance to 0835. By combining thepositions of the landmarks with the GPS data, a precise set of relativepositions can be calculated for landmarks in the video. The newlycreated two dimensional data for the location of the landmarks can nowbe added to a GIS viewing software application such as the open sourceUniversity of Minnesota MapSever.

FIG. 9 illustrates a process of frame position interpolation inaccordance with the invention. In the illustrated example, three linesegments are drawn showing three scenarios of road travel to illustratewhere large changes in velocity and direction within one secondintervals can cause major errors in calculations of Video Capture System0201 camera positions, if one is relying upon straight line travel frompoint to point. In a straight line travel 0901 three one second GPSpoints 0905, 0906 and 0907 are logged. A set of ten notch marksrepresenting every third video frame are illustrated as 0908 and 0909which are frames 4 and 7 respectively. In this case the actual cameraposition is nearly identical to the calculated camera position.

The second line segment 0902 illustrates the difference between actualtravel 0915 of the Video Capture System on a windy road with a stop sign0917 versus the calculated position of travel 0925 and 0925 versus thepoint to point travel 0926. The calculated position of travel is basedupon using a fourth order polynomial function with parameters that thecoordinates of four data points 0930, 0931, 0932 and 0933. This functioncalculates line segments illustrated as 0921, 0922, 0923 which havenotch marks. In the case of the first line segment 0921, it is drawnover the actual path illustrating a highly accurate estimation of thetravel path. In the second line segment 0921, a nearly straight line0922 is calculated versus the actual turn. In the worst case scenario ofline segment three 0925 there is a large deviation from the actual path0915 as a driver made a right turn at a stop sign (without stopping)within the one second interval. The calculation of 0925 is closer to theactual 0915 than the linear point to point 0926 line segment.

The third line segment 0903 illustrates the calculated path 0940 of aVideo Capture System as it navigates a cul-de-sac with is highlyaccurate in motion to the actual path versus the point to point path0941. A spline polynomial method works well in that it creates smoothcurves that fit the natural motion of drivers as they make constant Gforce turns with smooth turns on a steering wheel.

FIG. 10 illustrates an aerial transition to street process. A set offour illustrations 1010, 1020, 1030 and 1040 are provided, whichrepresent a single frame or portion of a window within an Internet webpage. The top three illustrations 1010, 1020 and 1030 are created usingan open source GIS Mapping program, such as University of Minnesota MapServer, to show parcel line boundaries. The three sequentialillustrations show the effect as a user is zooming twice from 1010 to1020 and then 1020 to 1030. Once the user has reached a designated zoomheight as in the case of web page view 1030 where individual parcels canbe clearly seen, addition image layers 1031, 1032 and 1033 are added.The centroid 1031 of a selected parcel is identified with the motion oftravel 1032 of the video segment as the starting point of Video CaptureSystem camera position, and direction of travel 0133 with the dashedline 0134 representing the field of view of the camera. Finally, thefourth web page frame 1040 in the sequence illustrates the smoothtransition from aerial to street view of a selected parcel. In this casea single frame is shown based on the position 1035 of the Video CaptureSystem. The centroid of the parcel of web page frame 1030 is illustratedas 1031 and in the fourth web page frame 1040 the same parcel centroidis now illustrated as 1041 providing the end user good visual cues tohelp them see where they are as they view a street level image. Theactual path of the Video Capture System is drawn on the video frames as1042 as the drives past the designated parcel.

FIG. 11 illustrates a process utilizing real property vectors inaccordance with the invention, by which images captured by a left facingand a left front facing camera in a Video Capture System can calculateand generate superimposed property lines on both video and exporteddigital image photos from the video drive-by data. In this illustration,an ortho view 1200 of the travel of a Video Capture System along a roadis shown with four discrete image frame capture points 1211, 1221, 1231and 1241. These image frame capture points are illustrated on the lefthand side as 1210, 1220, 1230 and 1240. In the case of camera position1241 where the car is approaching the subject property 1249 and 1209 thecamera has a view of the left side of the road 1203 and 1243 where theproperty line 1246, 1206 and 1226 can be calculated as described aboveand shown on a digital image photo. The second 1230, third 1220 andfourth 1210 digital image photos show the changing perspectives of theVideo Capture System camera as it moves past a subject property.

FIG. 12 illustrates a process to extract and create a point of interestdatabase that is encoded with GPS data in accordance with the invention.Using the Video and Data Server Farm Image processing cluster of servers0115 and the video frame to digital image photo extraction process 0620,any one of several dozen OCR programs can be run on the extracteddigital image photos to build a results file of recognized text. In thiscase, a Video Capture System's front left camera at position 1305 isviewing a sign 1310 and 1320 that contains recognizable text. This textdata is then written into a database that includes but is not limitedto: the text characters, size and color of the text, relative positionof the characters in the field of view and GPS coordinates of the videoframe. Exact position in latitude and longitude of the text of the signcan be located as well as the relative orientation.

FIG. 13 illustrates two ortho views 1400 and 1450 of a Video CaptureSystem as it drives in a direction of travel 1410 along a path 1415 pasta selected parcel where one of six possible cameras may be viewing asubject property. The hexagonal array 1405 indicates the range of viewfor each of the six cameras. The Video Capture System at position 1401has a subject property centroid 1420 in the field of view 1425 of theright front camera. The field of view can be easily calculated bydrawing a virtual line segment 1445 from the camera position 1401 to thecentroid 1420. As the vehicle moves along the path 1465 in the secondortho view 1450 the right camera field of view 1475 is now containingthe centroid 1470 of the subject property when the camera is in position1451, which is illustrated by the virtual line segment 1460.

FIG. 14 illustrates a process by which an authenticated end user 1503can submit data regarding a specific location using video drive-by dataas geo-coding reference tool. The figure has three columns: the leftcolumn represents a Video and Data Server Farm 1501; the middle columnrepresents the Internet 1502; and on the right column represents an enduser 1503. The process starts when the end user 1503 establishescredentials 1505 to authenticate with personal information such as theirname, address and email with a credit card number, social securitynumber, date of birth, drivers license number, etc. At that point theycan select their login and password and submit 1506 to a common andcommercially available registration application 1510 running on anApache web server. Once a new user is established 1512, the user canthen login 1515 and send a request to authenticate 1516. When that useris authenticated 1522, they are now granted access 1525 to the videoserver 1530 where the can do a series of queries 1526 (by street addressor clicking on a map as shown in FIG. 5 or 7) where they obtain results1532 in the form of video feeds.

When the user is ready to review and edit property data, they login andauthenticate with the password. Using one of several modes of selectinga property, they view a particular parcel or area with a picture of theproperty. The viewer application can show video with various informationsuch as parcel data or property lines illustrated on the property. Theuser can also select the video to present frames at a slower rate than30 frames per second to allow careful viewing of the imagery. At thatpoint, where the end user sees a property, they can annotate anylandmark by clicking on the image of a property (such as the roofline ofa house 0545 shown in FIG. 5). They can use the cursor to establish apoint on an image as a designated landmark. The user can then tag thesame landmark in one or more video frames to further clearly tag theimage designated landmark. By processing the relative location of theimage in the video frames as the vehicle moves, the exact latitude andlongitude coordinates can be determined (with an estimated errorcalculation included in the data). When the user has selected the pointwithin the image, that location data of the one or mouse clicks with a(point, line segment or shading of area or attachment of an icon) can besubmitted 1536 to the video server 1540 and submitted 1537 to thedatabase server 1550 for permanent storage and retrieval by anotheruser. The database includes individual record entries of each and everyannotated data set to generate a detailed change log. This log file caninclude but it is not limited to latitude and longitude of cameraposition, estimated accuracy of camera position at time of capture, timeof original video capture as well as end user annotation time.

FIG. 15 illustrates a process and visual references as to thecalculations required to extract a photograph from a street number. Theprocess starts with the loading of county parcel data that typicallyincludes latitude and longitude coordinates of vector line segments thatidentify the individual parcel lines of given property. Typical countyparcel records include street name and number as part of the parcelnumber that is supplied as a separate database with the parcel numberitself being the unique ID that is used to link street address tolatitude and longitude encoded parcel lines. This figure includes twodiagrams 1605 and 1610. The top diagram 1605 shows a horizontallystitched together set of frames of video 1636 and 1641 as the leftCamera is moving by a set of houses. The stitching process can beaccomplished by using any one of several dozen commercially availableprograms running on the video image server cluster 0115 after the videoframes have been extracted as indicated in FIG. 6. The stitching programwill be feed a set of digital image photos in this example Frame 1 1636Frame 34 1641, Frame 67 1646 and Frame 100 1651 are used to composite acomplete wide horizontal view of the street. Each second of videoproduces thirty individual image frames. Accordingly, in this example,3.3 seconds of video were used from frame 1 to frame 100. The framenumber to use when driving by properties is easily calculated by thespeed data from the change in position of the GPS data points. Usingtechniques described above, text can be superimposed 1625 onto thedigital image photo.

As illustrated in diagram 1610, an end user is using a web page viewer1615 to move a set of left 1621 and right 1620 control arrow button toselect the image that evenly frames a subject property in the field ofview. By providing a long horizontal picture view 1605 in a squarecomputer monitor viewer web page 1615 an end user can easily identifyproperties with large street numbers 1625 displayed as they are lookingto identify a property.

FIG. 16 illustrates a novel method of finding a house from a photographbased on the invention. FIG. 16 includes three columns: left column 1701represents a Video and Data Server Farm, the middle column representsthe Internet 1703; and the right column represents an end user. Theprocess starts with a user entering a web page 1705 where they canupload a digital image photograph 1706 of a property taken from thestreet. The user can also input a zip code where they think the propertyis located to enable faster searching. Once they have entered the zipand attached the photograph, they can submit 1710 the data to theserver. At the Video and Data Server Farm the uploaded image isconverted 1715 to the appropriate standard format of uncompressed bitsrepresenting each pixel. The uncompressed image is then processed withone of several commercially available pattern recognition applications1725, where the uploaded image 1706 is compared to the street leveldigital image photographs extracted (for example those illustrated inFIGS. 13 and 15) from the video drive-by process. The photos that havepatterns matching the closest to the submitted 1706 photo are thenextracted and added to a webserver to create a web page 1725. This webpage is then delivered to the end user as a web page 1735 with theoriginally selected photo 1706 redisplayed as 1740 with additionalextracted images 1745, 1750 below. The end user does the final humanbased pattern matching by selecting with a radio dialog 1755 the imagethat looks the closest and then presses the submit button 1760. Thisdata selection is sent to the server 1765 where the server processes theselection 1770 to generate a web page that returns 1775 the streetaddress and picture of the property 1780.

FIG. 17 illustrates one novel method of identifying a home from a largelibrary of home photos with a high speed process. The process involvesusing one of many commercially available applications that generate edgedetection patterns in an image. In this figure, an image of a house 1801is shown with various lines that would be trivial for an edge detectionalgorithm to find properly. In order to process the data mostefficiently, only horizontal and vertical edges would be selected. Inthis figure, horizontal lines such as the top of the front door 1815,top of the roof 1816, and roof line 1817 and vertical lines such asfront door 1811, left edge of the right window 1812 and right edge ofthe right window 1813 are easily identified. In the second diagram 1802the vertical edges are displayed as a set of bar codes that have arelative horizontal distance from each other. The bar line 1821 isderived from the right side of the front door 1811 and the left side ofthe right window creates another bar line 1822. In the third diagram1803, the horizontal edges that have been detected are displayed as barlines. In this diagram bar line 1836 is detected on the top of roof line1816, and bar line 1837 is the roof line 1817, and the bar line 1835 isthe top of the door 1815 and finally, the bar line 1836 is the bottom ofthe right window sill 1816. The storage of the discrete bar lines asmathematical values with relative position data enables a numericpattern match of values between photos taken at different distances froma subject property. In this way a set of numbers representing thelargest bar lines as vectors can be stored and compared against a largenumbers of properties with a simple sorting process.

The present invention may be applied to many applications in which theremote ability to view a specific location and its surroundingneighborhood is advantageous. For example, land owners, prospective landowners, insurance, mortgage, county appraisers need accurately todetermine where property lines are located in relation to a property.Today, a physical visit to a property with a survey in hand is requiredto start the process of determining where the property lines arelocated. In fact, many times a professional surveyor may be required tofind the original iron stakes that are usually covered over withvegetation. Once the stakes are found, using long tape measures andsurveying equipment, an accurate determination can be made of thelocations of various structures and landmarks in relation to theproperty lines. In contrast, the present invention permits the merger ofa unique set of data together to generate street level digital imagephotographs of properties that include superimposed illustrations ofproperty lines based on original latitude and longitude data provided bythe county property appraiser's office. This highly convenient method ofillustrating survey data fits a wide variety of unmet market needs.

As another example, occasions exist where people as they drive pastproperties want to find out who owns the property and what they paid forit. The present invention enables an end user the ability to select anystreet via an on-line web page and scroll along a view of images wherethe county parcel data is superimposed in the video stream. Images ofproperty parcels as well as the camera orientation pointed toward thecentroid of the parcel are provided along with property record data thatcan be processed to locate the centroid of the property and then displaytext data of sale price, owners name & street address above the image ofthe house.

Further, property appraisers are mandated by many state and localgovernments to obtain photographs of properties that they tax. Currentlysome counties pay $10 to $50 to have a driver go to specific properties,take a digital image photo and email it to the appraiser. Also, somecounties contract to have firms drive along streets shooting photos andstoring them in a database that they deliver to the appraiser for a feeof $1 to $2 per parcel. In contrast, with the present invention, acounty government could contract to view video of parcels in a far moreeconomical manner by using our Video Capture System driven by theirexisting appraisal staff. The data they capture can then be added to theVideo & Data Server Farm and is available for viewing in a 360° view notjust in a single view of a property. From the video, dozens ofindividual photos are available for the property appraiser to view atany time and by default the centroid oriented view at a 90 degree anglecan be shown first.

Still further, emergency service agencies have equipped their vehicleswith high speed wireless data networks to augment their voice radionetwork. The wireless data networks enable access to the Internet andthe Virtual Private Network to provide information related to thelocation and details on the emergency call while in route. This locationdata is typically a street address and a map with directions. Maps todaygenerally do not have good positional accuracy for individual streetaddresses. So when emergency responders pull onto the street, theyimmediately commence looking for house numbers. When these numberscannot be found especially at night, they resort to guessing and thisconsumes valuable time when seconds count. A solution which has beenimplemented in several counties is to provide a database of streetphotos that are tied to parcel data. This process is done by manuallyphotographing properties and linking them to a database of streetaddresses that may or may not be encoded with latitude and longitudedata from a GPS. The present invention enables the automatic andlow-cost capture of street level video which can be encoded with parceldata and GPS coordinates so that the Video & Data Server Farm cangenerate a long horizontal street level image that can be transmittedrapidly as a single image to an emergency vehicle in route which aids inthe ability for the emergency responder to quickly decide at which houseto stop.

Today, on-line map users are presented with a top view of a set ofstreets that are annotated with color highlighting and indicators suchas red stars that make it easy to identify where they need to go. Thepresent invention includes the additional opportunity to present streetlevel images of turns and intersections that enable users to “see” whatan intersection looks like a few hundred feet before they arrive andneed to turn. With the present invention, the user could see that thereis street level video available based on showing a double red line forexample that the user could click on to see a street level image. Whenthe user clicks on the red line, they would immediately be presentedwith video from that exact latitude and longitude coordinate.

Today, home buyers turn their heads as they drive by properties thathave for sale signs in the yard. Although there is no system on themarket today for viewing street level video on-line, there is a need forthe camera view to be oriented at a property as the 360 video capturesystem moves past the property. With the present invention, the Video &Data Server Farm has the latitude and longitude coordinate data of thecamera as it drives by and in addition, and it has the latitude andlongitude coordinate data of the centroid of the parcel that is forsale. Given this data it is easy to determine which camera position isactually pointed at and has in its field of view the centroid of theparcel. Typically homes are located with the foundation over thecentroid and thus the automatic camera orientation method provided bythe present invention will enable the automatic selection of the correctcamera view to present to the user when they are “driving online”.

Today as appraisers and real estate sales people insurers, construction,repair, maintenance people are interested in specifying a location or alandmark for a property, there is no facility available on line wherebya user could login annotate an image and then store that for viewing byothers. With the present invention, a user could access a video and/or aset of image files that include GPS coordinate data, camera orientationand optionally property line and parcel data superimposed over theimage. With this image available, the invention provides a simpleon-line viewing application that enables a user with his mouse to clickon the image presented and then open a text/dialog box where they canadd comments to the designated landmark within the photo. Given anauthentication process where the comments made by the individuals can bestored within the database, a permanent log can be made and viewed byothers relating to the property with accurate representation as to whomade the comment.

Today, when photographers visit a property to take a street levelphotograph, they naturally take care as to locate the camera along thestreet and orient the camera within a few degrees left and right as tothe field of view so the picture “looks good” and the house is “wellcentered”. The present system allows the automation of finding areasonably good view of a property, by utilizing the centroid image viewautomation process with an end user based process of “fine tuning” aview of a property. With the user viewing a stitched image of thesubject and its next door neighbors, they can slide the bar right andleft to simulate their movement of a photographer on the street movingright and left to get the image “just right”. Once that user selectstheir favorite frame, the system will store this optimal view coordinatedata and also generate a full resolution digital image photo file fordelivery to the user.

Today when people view a photograph of a home that they are interestedin buying on-line on a public website, they are not given the address ofthe property. Typically they are only provided with a photograph and thezip code of the property. In this regard a user is constrained fromdetermining if the property is in a neighborhood they like or is near aparticular set of amenities such as school, or church. For a buyer tolocate the house, they would need to invest time to contact either thelisting agent or they would need to agree to be represented by an agentto access the street address information. With the present invention,one could have an image of this home in the Video and Data Server Farm,thereby enabling a user to submit an image of the home to the Video andData Server Farm to process and locate. The servers would run one ofseveral commonly available pattern matching processes against the imagesfor a particular zip code and then present a set of properties that“look close” to the one the user submitted. Given the fact that humansare vastly superior to systems for pattern recognition, a list of 20close matches out of 2000 potential homes in a given zip code can beeasily viewed and selected by an end user in a couple of seconds. Oncethe user has selected the home that matches, they can then be presentedwith the street address and any other market comparable data they maywish to obtain.

In another embodiment, methods and apparatus are provided for presentinga three-dimensional model of a subject, such as a geographic area. Thethree-dimensional models can include a polygon based model formatgenerated from a point cloud array and sprayed with actual image data ofthe subject, captured from disparate points along a continuum.

A user interface is presented with user interactive controls that allowa user to traverse the three-dimensional model. A user location withinthe model will determine which portions of the three-dimensional modelare viewed. Some embodiments can include user interactive devices thatallow a user to traverse the three-dimensional model in threedirections: a) an essentially up and down vertical movement; b) anessentially side-to-side horizontal movement; and c) a depth movementinto and out of the three-dimensional model. Image data andthree-dimensional polygon based models presented to an inventor at agiven instance can be dependent upon the relative position selected bythe user within the user interface.

In another aspect, icons, overlays and metadata can be presented to auser upon demand to provide further information relating to a givengeographic area within view.

Still another aspect can include presentation of one or more geographicareas based upon criteria ascertainable from related data. For example,image data can be correlated with geospatial data so that a particularstreet address or Cartesian coordinate, such as a latitude/longitudelocation, is designated and specific image data is presented whichcorrelates with the geospatial data. The specific image data can bepresented sprayed over three-dimensional structures generated fromtwo-dimensional image data captured from that specific location.

In additional aspects, some embodiments can also include indications onthe user interface of user selectable data related to a particularlocation. User selectable data can include, for example: cellular phonesignal strength; wireless network data, including network name, locationand signal strength; global position system satellite data and signalstrength; radio signal strength; school district data, average yardsize, property price range; building features; proximity to amenities;political boundaries; zoning restrictions; and almost any othermeasurable quantity. Data can all be correlated according to ageographic location and presented in various combinations or upondemand. In addition, the data can be presented in conjunction with orindependent of, an image data continuum or image data set.

Definitions

As used herein, Video DriveBy™ refers to street level video datacaptured in multiple angles encompassing a 360° view.

As used herein, Video FlyBy™ refers to Aerial/Satellite oblique(angular) view images with polygon line views.

As used herein, RibbonView™ refers to a film strip like view ofproperties that stands up at approximately 90° from a flat or angledaerial/satellite ortho image map and provides direct-on front images ofproperties to be displayed.

As used herein, Flash Viewer (Streaming Video) refers to directstreaming of video to an online user via a web browser.

Methods

According to the present invention, image data is captured fromdisparate points along a continuum. Referring now to FIG. 18, in someembodiments the continuum 100 includes the path of a vehicle carrying adigital camera, or other image data capturing device. Image data setsare captured at disparate points 101-106 along the continuum. Somepreferred embodiments include capturing each image data set at an anglethat is generally orthogonal to the subject matter 107.

Positional data and orientation of the camera capturing image data setsis recorded for each disparate point 101-106 along the continuum fromwhich an image data set is captured. Positional data can include anydata indicative of where the subject matter of an image data set islocated. Some preferred embodiments of positional data include Cartesiancoordinates that are context sensitive according to the mechanism usedto generate coordinate data.

Positional data can be generated, for example, by an image datarecording device, such as a camera, associated with a device forrecording a global position (GPS device). Time stamps associated withimage data sets and time stamps associated with the GPS data can beutilized to correlate the GPS location data with image data set recordedby the image data recording device.

In still another aspect, in some embodiments, an altimeter can be usedto record an altitude from which a camera records image data sets. Thealtitude data can be associated with an image data set, for example,metadata correlated with the image data set. Such embodiments cantherefore include recordation of a latitude, longitude and altitudecoordinate for a given image data set. In addition, it is also withinthe scope of this invention to record a time of generation of an imagedata set and a depth of focus for an image data set.

According to the present invention, geospatial data, such as latitudeand longitude coordinates, can be generated by the GPS and stored withimage data generated by the camera. In some embodiments, GPS data can betime stamped and collected once every second. However, in someinstances, GPS reception can be interrupted, depending upon locationrelative to large object, such as multistory buildings, or cold cover.Therefore, some additional embodiments can include an accelerometer forrecording motion associated with a camera and a GPS unit operativelyattached to the camera.

Data from the accelerometer can indicate movement of the camera. Someaccelerometers, such as micro electromechanical system (MEMs)accelerometers can easily be incorporated into a camera system assemblyattached to a vehicle. Use of multiple MEM accelerometers positioned tomeasure movement in four or more directions along an x-axis, y-axis, andz-axis in relation to a camera can also be used to calculate directionof movement. The accelerometer can therefore be used to extrapolate acurrent position of the camera, based upon a last set of GPS geospatialdata recorded.

Geospatial data can be used to indicate an initial geographic position.A change in geospatial data can be additionally utilized to indicatevelocity and direction of image data set capture. Accelerometer data mayalso be used to indicate a velocity and direction of image data setcapture. Accelerometer data may also be used to indicate a velocity anddirection of camera movement. Calculations of time elapsed at theindicated velocity (such as for example, the Kalman Filter) can yield acalculated position at a time of image capture, even if the time ofimage capture is between GPS readings.

For example, one standard can include tracking a camera position with aGPS unit that records location at a rate of once per second. The cameracan record image data at a faster rate than once per second, such as,for example, one of: 12 images per second, 24 images per second or 29.97images per second. An initial camera position can be recorded thatcorrelates with a GPS reading; subsequent image data capture will occurin between the one second GPS reading interval. The camera position canbe determined with a calculation based upon the velocity of cameramovement supplied by the accelerometer and time elapsed since a previousGPS reading.

Still other embodiments can utilize optical flow methodology and visualodometry to facilitate calculations of a camera position and thevelocity of a vehicle or person from which a series of image data setsare captured. Visual odometry can be accomplished with a singleomni-directional camera or with stereo cameras and is based uponprocessing that tracks the position of salient features in a series offeature sets and calculates movement based upon the relative positionsof the features. In some embodiments, camera based simultaneouslocalization and mapping (SLAM) of visual image data can also be used tofacilitate calculations of velocity of a change in position of a cameraused to capture image data sets. Typically, the velocity will bedirectly tied to the motion of a vehicle to which the camera is mounted,or a person carrying a camera rig.

Orientation of a camera can include a direction of image capturerecorded by the camera. Orientation can be designated, for example, inrelation to the cardinal directions, i.e., north, south, east and west.Any means available to record such a designation, such as an electroniccompass, is within the scope of the present invention. However, it maybe desirable to include a means to record the orientation with a greaterdegree of accuracy than is typically available through the use of anelectronic compass.

Therefore, in some embodiments, orientation can be determined accordingto a fixed position of a camera in relation to the direction of travelof a vehicle (or person) used to transport the camera. For example, aplurality of cameras can be fixedly attached to a vehicle capturingVideo DriveBy™ data. Each camera therefore maintains a constantdirection of image capture in relation to the heading of the vehicle.Mechanics of the camera, such as, for example, lens parameters andshutter speed, can indicate a depth of field during camera image datacapture. Some embodiments can also include simultaneously capturingmultiple image data sets and correlating two or more of the image datasets. Correlation can be accomplished via a time stamp or otherchronological or synchronous.

The position of a camera can be combined with a direction of imagecapture and the depth of field of the camera to determine a location ofimage data captured by the camera at a particular instance in time. Thepresent invention can also include apparatus for utilizing echo locationto determine a distance of an object from a camera capturing an imagedata set and storing a correlated distance with the image data set. Forexample, radar data can be correlated with a camera image data set tocalculate the location of various objects captured by the camera. A timestamp can also be combined with data to quantify a location for aparticular image formed by the captured image data.

In some embodiments of the present invention data used to calculate alocation of an image is stored in a metadata file space associated withthe image data. For example, some embodiments can store metadata in theexchangeable image file format (EXIF), TIFFTAGS or International PressTelecommunication Council (IPTC) formats. Image data may be stored, forexample in JPEG or TIFF formats. However, other metadata formats canalso be used. Typically, due to the size of data files that aregenerated during capture of Video DriveBy™ data, the image data andmetafile data are stored on an external data storage device, such as ahard disk drive operatively attached to the camera. However, in someembodiments, the data can be stored in the camera.

As discussed above, the metadata can include data descriptive of some orall of: date and time; camera settings such aperture, shutter speed andfocal length; geospatial data from a GPS receiver unit; accelerometerdata; inertial guidance system data; camera orientation; and camerafixed position related to vehicle travel.

Referring now to FIG. 19, exemplary image data sets 201-203 areillustrated which capture images of a subject 200 from different pointsalong a continuum. In some embodiments, as illustrated in FIG. 20, imagedata sets 304-307 overlap the subject matter captured. The overlapallows for features present in one image data set to be present in asecond image data set, and preferably in a tertiary image data set.

Referring now to FIG. 21 a block diagram illustrates three-dimensionalstructures 405-407 created from image data captured. In someembodiments, the structures can be created by generating a point cloudarray from a plurality of features contained in the image data sets andconverting the point cloud array to a polygon based model. The polygonbased model can be associated with a location relative to the image dataset and image data can be sprayed over the polygon based model. In somepreferred embodiments, image data sprayed over the polygon based modelsincludes the composite of image data.

According to the present invention, a user can traverse the presentationof the composite of image data and the three-dimensional structures.Movement through the user interface can be accomplished with any knowninteraction device, such as, for example one or more of: a keyboard,mouse, video game controller Apple I Phone, digital cellular phone,remote controller, WiiMote and a joystick.

The user interface can allow a user to move in three directions througha scene presented. The directions can include an x coordinate providinga vertical dimension 401, a y dimension providing a horizontal dimension402 and a z coordinate providing a depth dimension 403.Three-dimensional modeling sprayed with actual image data allows theuser to proceed in “off-track” paths that do not match a path created bythe disparate points from which data is collected.

In some embodiments, a time dimension can also be traversed, such as,for example, image data sets of a geospatial designation across timechronology such as a period of seasons or years. Some embodiments caninclude a “take me there” time designation, wherein a user can indicatea time period and geospatial description. According to the presentinvention, a three-dimensional model can be generated with image datasets from the indicated time period for the designated geospatial area.

As the user moves into a scene presented in the user interface, planesof image data 405 can change according to the perspective chosen by theuser. For example, a three-dimensional structure of a house can besprayed with image data included in the continuum composite of imagedata. The actual image data will correlate with a surface of thethree-dimensional structure facing the disparate points from which theimage data was captured. According to the present invention, additionalsurface of the house 406 can also be sprayed with a texture and colorcaptured by the actual image data. In this way, as a user moves in a zdirection beyond front surface of the house, the user will be presentedwith additional surfaces of the house, with each surface having a colorand texture extrapolated from the image data from the front of thehouse.

In addition, an object such as a tree 405 can also be created in threedimensions with color and texture based upon the image data composite.In some embodiments, user interactive device controls can be providedwhich allow a user to remove objects such as the tree 405 and the house406. Removal of an object will allow the user to see beyond the removedobject into the scene provided. For example, if a tree 405 obscures theview of the front of a house 406, activation of a user interactivecontrol can remove the tree 405 allowing an unobstructed view of thethree-dimensional model of the house 406. If actual image data is notavailable for those portions of the house now viewable in theunobstructed view, then the present invention allows for color andtexture data matching the known surfaces of the house to be sprayed overthose portions now viewable.

Some embodiments can additionally allow a user to modify color andtexture of three-dimensional models presented to them. Suchmodifications facilitate “what-if” scenarios. For example, a buyercontemplating purchasing a particular property can be presented with athree-dimensional model of the property with three-dimensionalrepresentations of objects on the property sprayed with actual imagedata captured of the property. According to some embodiments, the buyercan then modify the view presented to represent possible changes thatmay be made to the property. Changes may include, to continue theexample, providing a brick facade over a painted surface, or removal ofshrubbery, or addition of shrubbery. Addition of items can be providedfrom a menu of three-dimensional objects made available to the user.Color selections and textures can also be made available via a userselectable menu. In this fashion a unique blend of a composite continuumof image data and user defined image data can be presented, includingthree-dimensional modeled objects derived from the image data and usermodifications to such models.

By way of non-limiting example, in some embodiments, a user may be ableto select a building and designate from a user interface a type andcolor of roofing the user would like to see illustrated for thebuilding. Accordingly, a user may designate an architectural shingle invarious hues, or a Spanish tile or steel roof and the present inventionwill spray the pattern and color selected onto a three-dimensional modelof the building. In still other embodiments, it is within the scope ofsome embodiments of the present invention to allow a user to upload atexture and color to be sprayed upon a three-dimensional model, or aportion of the three-dimensional model, such as a roof portion.According to such embodiments, a user may upload a color or hue ortexture to be sprayed upon a three-dimensional model of a structure. Thecolor and hue may also be stored and made available for subsequentapplication to portions of an image data set.

Referring now to FIG. 22, the present invention can also includegeneration of a user interface 500 with multiple representations ofimage data relating to a geographic area. For example, an aerial view ofa geographic area 504 can be combined with one or more two-dimensionalcontinuums of composite image data 505-506. The two-dimensionalcontinuums of composite image data 505-506 can be generated according tomethods and apparatus described in a related co-pending patentapplication by the same inventors. In addition, one or more continuumsof disparate points from which image data is captured can also beindicated on the aerial view 504. Three-dimensional models 510-503 canalso be included in the user interface 500 and located on the aerialview according to geospatial data generated during image data captureand processing of the image data.

As described in a related application, the three-dimensional models canbe generated from multiple image data sets captured from disparatepoints along a continuum sprayed with the image data captured.

According to some embodiments of the present invention, a user maylocate a particular geographic area via different vehicles madeavailable to a user. For example, a user can enter a geospatialdesignation, such as, for example, one or more of: a street address; aCartesian coordinate; a “Favorite” name designating a geospatial area; apopular name for a geospatial area; a coded indication for a geospatialarea, such as a multiple listing number or data file number; andmetadata. According to the present invention, a “take me there” commandcan be executed to bring the user to the requested geospatial area.Alternatively, the present invention provides a vehicle “cockpit”interface which allows a user to virtually “fly” over an aerial view andthen transition into a virtual “dashboard” and “drive” mode along astreet level two-dimensional continuum of actual image data, and in somepreferred embodiments, a continuum of composite image data. The user mayalso transition into a virtual three-dimensional modality which includesthree-dimensional models sprayed with image data captured from acontinuum of disparate points. The three-dimensional models can beviewed from a virtual “drive by” mode or a virtual “walkabout.” Thevirtual walkabout can allow a user to go off path and view thegeographic area from amongst three-dimensional models sprayed with imagedata captured from a continuum of disparate points. In addition, asdiscussed above, the user may manipulate the appearance of thethree-dimensional models.

A take me there command can be additionally operative to designate amodality, such as the aerial view, two-dimensional ribbon view, orthree-dimensional drive by or three-dimensional walk about view intowhich a user will enter.

In another aspect of the present invention, a user can be provided withtools that provide analysis of the image data shown. Analysis caninclude, for example, designation of a particular portion of ageographic area 508. The portion 508 can be representative of a realestate parcel, as indicated by municipal records, or some otherdesignation. For example, a user may draw a polygon 508 onto an aerialimage of a geographic area 500 and ask for analysis of the areacontained within the polygon 508.

Analysis can include, by way of example, a calculation of an areaincluded within the polygon. Other analysis can include moresophisticated calculations. For example, it is within the scope of theinvention to designate property boundaries, such as, through countyrecord data and locate an image data of the property. The image data canbe used to generate a three-dimensional model of the property, includinga model of a house within the property boundaries and other features,such as trees and utility demarcations. Based upon the location of thehouse within the property boundaries, the present invention may also beused to generate a size of a front yard, and, by extrapolation, the sizeof the side and back yards. An approximate size of a house can also becalculated based the dimensions of the three-dimensional modelgenerated. Other features, such as how many stories, the house includesand the type of facade the house has may also be ascertained fromanalyzing the image data and the model.

Statistical representations can also be made, according to the analysis.Statistical representations can provide a summary of characteristics ofa single real estate parcel, or a designated area. By way of example,statistical analysis can include one or more of an average, median andhigh/low value designation for: the size of a property within a givenneighborhood, the size of a front yard, how much road frontage eachproperty includes, the average size of the houses within the designatedarea, the number of stories comprising the buildings within the area,siding types of buildings within the area, the size of side yardsbetween houses, height of trees within the area, and almost any otherdata that may be calculated from the image data sets and thethree-dimensional models.

Referring now to FIG. 23, a view of an alternative exemplary userinterface 600 according to the present invention is illustrated. Theinterface 600 can include a portion with an aerial view 601 and apolygon designation of geographic area of interest, such as for example,a particular real estate parcel or neighborhood location. The interfacemay also include a portion with a two-dimensional continuum of imagedata 603A. The continuum of image data is preferably a composite ofimage data sets captured from disparate points. Another portion 603B caninclude image data sprayed over three-dimensional models 604 generatedfrom point clouds of features captured by the image data sets.

Referring now to FIG. 24, in some preferred embodiments, image datasprayed over the three-dimensional polygon formats includes a compositeimage formed by aligning two or more of the image data sets. Unlikestitching processes previously known, the present invention creates acomposite through alignment of portions of data from more than one dataset. Alignment can be accomplished in image data processing. Using imagedata processing, the images 701-703 are aligned to form a compositeimage 700. The composite image 700 is essentially two dimensional imagedata arranged as a second continuum, or ribbon. The second continuumincludes ongoing image data 701-703 captured from the points definingthe first continuum.

In some particular embodiments, the series of points of image capture inthe first continuum includes positions of a vehicle carrying an imagecapture device, such as a camera, as the vehicle traverses a pathproximate to a geographic area. The camera is positioned to captureimage data of the geographic area. Image data 701-703 is periodicallycaptured as the vehicle traverses the path. The motion of the vehicle,combined with the periodic capture of image data 701-703, therebyresults in image data 701-703 being captured from disparate points alongthe first continuum.

A preferred embodiment includes capture of image data with a motionvector of the camera in space maintained generally orthogonal to asubject for which image data will be captured. Orthogonal capture of theimage data can facilitate consistency for subsequent composite ofportions of the image data captured. Therefore, data captured at anangle of between about 75° and 105° can provide data most easilyassembled into a continuum of data. However, other angles may be used tocreate different effects. In addition, in some embodiments physicallimitation may dictate the use of other angles. In such instances, postprocessing can be utilized to compensate for the angle of image datacapture.

During image data processing, some or all of the images are aligned toform a composite image in the form of a continuous pictorialrepresentation of the geographic area. One commercial embodiment of acontinuous pictorial representation includes RibbonView™ by Real DataSystems. RibbonView™ correlates a ribbon of geographic image data withgeospatial designations to facilitate identification of a particulargeographic area, as discussed more fully below. In various embodiments,image capture processing can be accomplished in real time or via postimage capture processing.

In some embodiments, select portions 704-707 of two or more sets ofcaptured image data are aligned to generate the composite image 708. Asillustrated, some preferred embodiments include vertical slices of data704-707 aligned in a horizontal plane to form the composite image 708.Unlike a traditional photograph taken with a macro lens, according tothe present invention, the length of a horizontal plane defining acomposite image 708 is only limited by the length of a continuum alongwhich points are defined and from which image data 704-707 is captured.

The use of only slices of data 704-707 from any particular capturedimage provides for a higher quality image 708. The quality is increased,for example, when a temporary obstruction, such as a passing car, personor animal, captured in one image data set, is only represented in a thinslice of a continuous ribbon 704-707 data. In addition, alignment ofmultiple thin slices of image data 704-707 is facilitated from theperspective of which aberrations typical human sensory is capable ofdistinguishing. Numerous thin slices 704-707 are perceived as a smoothertransition across the horizontal plane defining the composite image 708removing a choppiness that may be experienced with the use of largerdata images.

The present invention can include a uniform width of each slice of data704-707 or a variable width. The width of a particular slice may vary,for example, upon one or more of the velocity of a vehicle from whichimage data sets are captured, the sample rate of a camera used tocapture an image data set 701-703, the resolution of a picturecomprising an image data set 701-703 and the path of a camera. A highresolution image generated by a 2.1 mega pixel camera may have a 1600 by1200 resolution and allow for a thinner slice 704-707 that includes awidth of between about 5 to 700 pixels of an image data set. Embodimentswith very high quality can include a slice 704-707 of between 1 to 2pixels.

From a different perspective, some embodiments can include a slice704-707 of an image data set 701-703 that includes a percentage of theimage data set 701-703. Accordingly, some preferred embodiments caninclude a slice 704-707 of between about 5% to about 12% of an imagedata set. Other preferred embodiments can include up to about 50% of animage data set. However, it should be understood that some embodimentsinclude a slice 704-707 that constitutes any fraction of the completeimage data set.

It should be noted that although preferred embodiments may utilizevertical rectangular slices 704-709, the scope of the present inventionis not limited by which portion of an image data set 701-703 is utilizedto generate a composite image 270. Therefore, it is within the scope ofthe invention to use any portion of any image data set 701-703 togenerate a composite image. Accordingly, slices of an image 701-703other than vertical slices 704-709 may be apportioned and combined intoa composite image 270. Slices may therefore include a slice angled at60° or 75°, or other angle conducive to a particular application. Inaddition, it is also within the scope of the present invention toutilize irregular shaped portions of two or more image data sets 701-703to generate a composite image 270.

In some embodiments, a database or other data processing mechanism, cantrack each portion or slice 704-708 utilized to construct a continuumand associate the slice 704-708 with an original image 701-703 fromwhich the slice 704-708 is derived. User interactive devices can executethe retrieval of an entire original image 701-703 or a series oforiginal images 701-703 upon request. In some instances, one or moreoriginal images 701-703 may provide detail not included in the compositeimage 708.

In some embodiments, a selected portion of an image data set may bedetermined by the physical attributes of the equipment used to capturean image data set. For example, a typical camera lens can impart somedistortion to an image data set, as represented in the illustration byan elongated portion and a compressed portion. Utilizing only a portionof an image data set, such as, for example, a center portion verticalslice, can minimize the effect of distortion introduced by a lens, orother source, to a composite image. Distortion is minimized when thecomposite image is made to include those portions of the image data setcorresponding with the center of the lens. Specific embodiments maylikewise account for other aberrations that may be present in a set ofimage data.

It will be apparent to those schooled in the art that the length of acomposite image generated according to the present invention is limitedonly by the ability to capture image data from additional points on acontinuum and store the captured image data for post processing. Thepost processing allows for the alignment of portions of the image datacompiled into a composite two-dimensional view that can continue so longas additional image data is made available to be added to it.

Apparatus

The teachings of the present invention may be implemented with anyapparatus capable of embodying the innovative concepts described herein.Image capture can be accomplished, for example, via a digital cameracapable of capturing 12 or more images per second. In addition, FIG. 25illustrates a controller 800 that may be utilized to implement someembodiments of the present invention. The controller 800 comprises aprocessor unit 810, such as one or more processors, coupled to acommunication device 820 configured to communicate via a communicationnetwork (not shown in FIG. 25). The communication device 820 may be usedto communicate, for example, with one or more online devices, such as apersonal computer, laptop or a handheld device.

The processor 810 is also in communication with a storage device 830.The storage device 830 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g.,magnetic tape and hard disk drives), optical storage devices, and/orsemiconductor memory devices such as Random Access Memory (RAM) devicesand Read Only Memory (ROM) devices.

The storage device 830 can store a program 840 for controlling theprocessor 810. The processor 810 performs instructions of the program840, and thereby operates in accordance with the present invention. Theprocessor 810 may also cause the communication device 820 to transmitinformation, including, in some instances, control commands to operateapparatus to implement the processes described above. The storage device830 can additionally store related data in a database 830A and database830B, as needed.

In some preferred embodiments, apparatus includes a video and dataserver farm. The video and data server farm includes at least one videostorage server that stores video image files containing video drive-bydata that corresponds to a geographic location, a database server thatprocesses a data query received from a user over the Internet thatcorresponds to a geographic location of interest, and an image server.In operation, the database server identifies video image files stored inthe video storage server that correspond to the geographic location ofinterest contained in the data query, and transfers the video imagefiles over a pre-processing network to the image processing server. Theimage processing server converts the video drive-by data topost-processed video data corresponding to a desired image format, andtransfers the post-processed video data via post-processing network tothe Internet response to the query.

A landing zone server can also be included which receives the videodrive-by data from a portable memory device and permits the viewing andanalysis of the video drive-by data prior to storage in the videostorage server. Still further, a map server is preferably provided topresent a static image an overhead view of the geographic location ofinterest.

Embodiments can also include one or more servers described aboveincluded in one or more physical units. Each server does not need to bea disparate apparatus. Still other embodiments can include one or moreor the servers described above included in multiple physical units. Someembodiments can even include a single server, as described whichincludes multiple physical apparatus units at disparate locations.

Referring now to FIG. 26, three images 101-103 are illustrated. Each ofthe images 101-103 is reproduced from image data captured from adisparate point on a continuum. As illustrated, a composite image 100 isformed by aligning two or more of the image data sets. Unlike stitchingprocesses previously known, the present invention creates a compositethrough alignment of portions of data from more than one data set.Alignment can be accomplished in image data processing. Using image dataprocessing, the images 101-103 are aligned to form a composite image100. The composite image 100 is essentially two dimensional image dataarranged as a second continuum, or ribbon. The second continuum includesongoing image data 101-103 captured from the points defining the firstcontinuum.

In some particular embodiments, the series of points of image capture inthe first continuum includes positions of a vehicle carrying an imagecapture device, such as a camera, as the vehicle traverses a pathproximate to a geographic area. The camera is positioned to captureimage data of the geographic area. Image data 101-103 is periodicallycaptured as the vehicle traverses the path. The motion of the vehicle,combined with the periodic capture of image data 101-103, therebyresults in image data 101-103 being captured from disparate points alongthe first continuum.

A preferred embodiment includes capture of image data with a cameramaintained orthogonal to the subject. Orthogonal capture of the imagedata provides consistency for subsequent composite of portions of theimage data captured. Therefore, data captured at an angle of betweenabout 75° and 105° can provide data most easily assembled into acontinuum of data. However, other angles may be used to create differenteffects.

During image data processing, some or all of the images are aligned toform a composite image in the form of a continuous pictorialrepresentation of the geographic area. One commercial embodiment of acontinuous pictorial representation includes RibbonView™ by Real DataSystems. RibbonView™ correlates a ribbon of geographic image data withgeospatial designations to facilitate identification of a particulargeographic area, as discussed more fully below. In various embodiments,image capture processing can be accomplished in real time or via postimage capture processing.

Referring now to FIG. 27, in some embodiments, select portions 104-107of two or more sets of captured image data are aligned to generate thecomposite image 108. As illustrated, some preferred embodiments includevertical slices of data 104-107 aligned in a horizontal plane to formthe composite image 108. Unlike a traditional photograph taken with amacro lens, according to the present invention, the length of ahorizontal plane defining a composite image 108 is only limited by thelength of a continuum along which points are defined and from whichimage data 104-107 is captured.

The use of only slices of data 104-107 from any particular capturedimage provides for a higher quality image 108. The quality is increased,for example, when a temporary obstruction, such as a passing car, personor animal, captured in one image data set, is only represented in a thinslice of a continuous ribbon 104-107 data. In addition, alignment ofmultiple thin slices of image data 104-107 is facilitated from theperspective of which aberrations typical human sensory is capable ofdistinguishing. Numerous thin slices 104-107 are perceived as a smoothertransition across the horizontal plane defining the composite image 108removing a choppiness that may be experienced with the use of largerdata images.

The present invention can include a uniform width of each slice of data104-107 or a variable width. The width of a particular slice may vary,for example, upon one or more of the velocity of a vehicle from whichimage data sets are captured, the sample rate of a camera used tocapture an image data set 101-103, the resolution of a picturecomprising an image data set 101-103 and the path of a camera. A highresolution image generated by a 2.1 mega pixel camera may have a 1600 by1200 resolution and allow for a thinner slice 104-107 that includes awidth of between about 5 to 100 pixels of an image data set. Embodimentswith very high quality can include a slice 104-107 of between 1 to 2pixels.

From a different perspective, some embodiments can include a slice104-107 of an image data set 101-103 that includes a percentage of theimage data set 101-103. Accordingly, some preferred embodiments caninclude a slice 104-107 of between about 5% to about 12% of an imagedata set. Other preferred embodiments can include up to about 50% of animage data set. However, it should be understood that some embodimentsinclude a slice 104-107 that constitutes any fraction of the completeimage data set.

Referring now to FIG. 28, three sets of image data 201-203 areillustrated, wherein each set of image data 201-203 represents datacaptured from a disparate point on a continuum proximate to thelandscape captured. A direction of travel along the continuum 200 isalso illustrated. Each data set 201-203 is divided into multiple slicesof image data 204-209. A composite image 210 is generated from themultiple slices 204-209, and in the exemplary case at hand, additionalslices from additional image data sets.

It should be noted that although preferred embodiments may utilizevertical rectangular slices 204-209, the scope of the present inventionis not limited by which portion of an image data set 201-203 is utilizedto generate a composite image 210. Therefore, it is within the scope ofthe invention to use any portion of any image data set 201-203 togenerate a composite image. Accordingly, slices of an image 201-203other than vertical slices 204-209 may be apportioned and combined intoa composite image 210. Slices may therefore include a slice angled at60° or 75°, or other angle conducive to a particular application. Inaddition, it is also within the scope of the present invention toutilize irregular shaped portions of two or more image data sets 201-203to generate a composite image 210.

In some embodiments, a database or other data processing mechanism, cantrack each portion or slice 204-208 utilized to construct a continuumand associate the slice 204-208 with an original image 201-203 fromwhich the slice 204-208 is derived. User interactive devices can executethe retrieval of an entire original image 201-203 or a series oforiginal images 201-203 upon request. In some instances, one or moreoriginal images 201-203 may provide detail not included in the compositeimage 208.

Referring now to FIG. 29, in some embodiments, a selected portion 303 ofan image data set 300 may be determined by the physical attributes ofthe equipment used to capture an image data set. For example, a typicalcamera lens can impart some distortion to an image data set, asrepresented in the illustration by an elongated portion 301 and acompressed portion 302. Utilizing only a portion 302 of an image dataset 300, such as, for example, a center portion vertical slice 303, canminimize the effect of distortion introduced by a lens, or other source,to a composite image 108. Distortion is minimized when the compositeimage 108 is made to include those portions of the image data setcorresponding with the center of the lens 303. Specific embodiments maylikewise account for other aberrations that may be present in a set ofimage data.

In another aspect of the invention, and referring now to FIG. 30,positional data descriptive of a location of the subject matter of animage can also be generated by the image data recording device.Positional data can include any data indicative of where the subjectmatter of an image is located. Some preferred embodiments can includeCartesian coordinates that are context sensitive according to themechanism used to generate coordinate data.

For example, an image recording device, such as a camera, can beassociated with a device for recording a global position, such as aglobal positioning system (GPS) device or other equipment. Time stampsassociated with image data and time stamps associated with the GPS datacan be utilized to correlate the GPS location data with image datarecorded by the camera.

In still another aspect, in some embodiments, an altimeter can be usedto record an altitude from which a camera records image data sets. Thealtitude data can be associated with an image data set, for example,metadata correlated with the image data set. Such embodiments cantherefore include recordation of a latitude, longitude and altitudecoordinate for a given image data set. In addition, it is also withinthe scope of this invention to record a time of generation of an imagedata set and a depth of focus for an image data set.

According to the present invention, geospatial data, such as latitudeand longitude coordinates, can be generated by the GPS and stored withimage data generated by the camera. In some embodiments, GPS data can betime stamped and collected once every second. However, in someinstances, GPS reception can be interrupted, depending upon locationrelative to large object, such as multistory buildings, or cold cover.Therefore, some additional embodiments can include an accelerometer forrecording motion associated with a camera and a GPS unit operativelyattached to the camera.

Data from the accelerometer can indicate movement of the camera. Someaccelerometers, such as micro electromechanical system (MEMs)accelerometers can easily be incorporated into a camera system assemblyattached to a vehicle. Use of multiple MEM accelerometers positioned tomeasure movement in four or more directions along an x-axis, y-axis, andz-axis in relation to a camera can also be used to calculate directionof movement. The accelerometer can therefore be used to extrapolate acurrent position of the camera, based upon a last set of GPS geospatialdata recorded.

Geospatial data can be used to indicate an initial geographic position.A change in geospatial data can be additionally utilized to indicatevelocity and direction of image data set capture. Accelerometer data mayalso be used to indicate a velocity and direction of image data setcapture. Accelerometer data may also be used to indicate a velocity anddirection of camera movement. Calculations of time elapsed at theindicated velocity (such as for example, the Kalman Filter) can yield acalculated position at a time of image capture, even if the time ofimage capture is between GPS readings.

For example, one standard can include tracking a camera position with aGPS unit that records location at a rate of once per second. The cameracan record image data at a faster rate than once per second, such as,for example, one of: 12 images per second, 24 images per second or 29.97images per second. An initial camera position can be recorded whichcorrelates with a GPS reading, subsequent image data capture will occurin between the one second GPS reading interval. The camera position canbe determined with a calculation based upon the velocity of cameramovement supplied by the accelerometer and time elapsed since a previousGPS reading.

Still other embodiments can utilize optical flow methodology and visualodometry to facilitate calculations of a camera position and thevelocity of a vehicle or person from which a series of image data setsare captured. Visual odometry can be accomplished with a singleomni-directional camera or with stereo cameras, and is based uponprocessing which tracks the position of salient features in a series offeature sets and calculates movement based upon the relative positionsof the features. In some embodiments, camera based simultaneouslocalization and mapping (SLAM) of visual image data can also be used tofacilitate calculations of velocity of a change in position of a cameraused to capture image data sets. Typically, the velocity will bedirectly tied to the motion of a vehicle to which the camera is mounted,or a person carrying a camera rig.

Orientation of a camera can include a direction of image capturerecorded by the camera. Orientation can be designated, for example, inrelation to the cardinal directions, i.e. north, south, east and west.Any means available to record such a designation, such as an electroniccompass, is within the scope of the present invention. However, it maybe desirable to include a means to record the orientation with a greaterdegree of accuracy than is typically available through the use of anelectronic compass.

Therefore, in some embodiments, orientation can be determined accordingto a fixed position of a camera in relation to the direction of travelof a vehicle (or person) used to transport the camera. For example, aplurality of cameras can be fixedly attached to a vehicle capturingVideo DriveBy™ data. Each camera therefore maintains a constantdirection of image capture in relation to the heading of the vehicle.Mechanics of the camera, such as, for example, lens parameters andshutter speed, can indicate a depth of field during camera image datacapture. Some embodiments can also include simultaneously capturingmultiple image data sets and correlating two or more of the image datasets. Correlation can be accomplished via a time stamp or otherchronological or synchronous.

The position of a camera can be combined with a direction of imagecapture and the depth of field of the camera, to determine a location ofimage data captured by the camera at a particular instance in time. Thepresent invention can also include apparatus for utilizing echo locationto determine a distance of an object from a camera capturing an imagedata set and storing a correlated distance with the image data set. Forexample, radar data can be correlated with a camera image data set tocalculate the location of various objects captured by the camera. A timestamp can also be combined with data to quantify a location for aparticular image formed by the captured image data.

In some embodiments of the present invention data used to calculate alocation of an image is stored in a metadata file space associated withthe image data. For example, some embodiments can store metadata in theexchangeable image file format (EXIF), TIFFTAGS or International PressTelecommunication Council (IPTC) formats. Image data may be stored, forexample in JPEG or TIFF formats. However, other metadata formats canalso be used. Typically, due to the size of data files that aregenerated during capture of Video DriveBy™ data, the image data andmetafile data are stored on an external data storage device, such as ahard disk drive operatively attached to the camera. However, in someembodiments, the data can be stored in the camera.

As discussed above, the metadata can include data descriptive of some orall of: date and time; camera settings such aperture, shutter speed andfocal length; geospatial data from a GPS receiver unit; accelerometerdata; inertial guidance system data; camera orientation; and camerafixed position related to vehicle travel.

Referring now to FIG. 31, an illustration of some RibbonView™embodiments of the present invention is illustrated. The RibbonView™illustrated 501-502 includes strips or ribbons of two-dimensional imagedata. Each strip 501-502 is generated via the capture of image dataalong a continuum 503. As illustrated, the continuum 503 generallyfollows the roadway in the center of the illustration. Each compositeimage 501-502 is generated through the alignment of a portion ofmultiple images captured as data from disparate points along thecontinuum 503. In some preferred embodiments, the present inventiondelivers two ribbons of composite images 501-502 with different viewscaptured along the continuum. Each view is generated by a camera with aseparate orientation.

It will be apparent to those schooled in the art that the length of acomposite image generated according to the present invention is limitedonly by the ability to capture image data from additional points on acontinuum and store the captured image data for post processing. Thepost processing allows for the alignment of portions of the image datacompiled into a composite two-dimensional view that can continue so longas additional image data is made available to be added to it.

Apparatus

The teachings of the present invention may be implemented with anyapparatus capable of embodying the innovative concepts described herein.Image capture can be accomplished, for example, via a digital cameracapable of capturing 12 or more images per second. In addition, FIG. 32illustrates a controller 600 that may be utilized to implement someembodiments of the present invention. The controller 600 comprises aprocessor unit 610, such as one or more processors, coupled to acommunication device 620 configured to communicate via a communicationnetwork (not shown in FIG. 32). The communication device 620 may be usedto communicate, for example, with one or more online devices, such as apersonal computer, laptop or a handheld device.

The processor 610 is also in communication with a storage device 630.The storage device 630 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g.,magnetic tape and hard disk drives), optical storage devices, and/orsemiconductor memory devices such as Random Access Memory (RAM) devicesand Read Only Memory (ROM) devices.

The storage device 630 can store a program 640 for controlling theprocessor 610. The processor 610 performs instructions of the program640, and thereby operates in accordance with the present invention. Theprocessor 610 may also cause the communication device 620 to transmitinformation, including, in some instances, control commands to operateapparatus to implement the processes described above. The storage device630 can additionally store related data in a database 630A and database630B, as needed.

CONCLUSION

The invention has been described with reference to certain preferredembodiments thereof. It will be understood, however, that modificationsand variations are possible within the scope of the appended claims. Forexample, various methods or equipment may be used to implement theprocess steps described herein or to create a device according to theinventive concepts provided above and further described in the claims.In addition, various integration of components, as well as software andfirmware can be implemented.

For example, while the illustrated embodiment utilizes a camera arraycontaining analog cameras requiring digital conversation, the cameraarray may be implemented with cameras that directly produce a digitaloutput. Also, digital cameras may be employed instead of analog cameraswith separate A/D converters. The number of cameras in the array mayalso vary. Further, while the preferred embodiment utilizes externalhard disk drives, it will be understood that any type of portable memorydevice having sufficient capacity to store the video data may beutilized. Still further, it will be understood that the variousprocessing methods may be implemented utilizing the servers of the Videoand Data Server Farm, the processing unit, i.e. the laptop computer, ofthe Video Capture System, and/or other processors linked thereto throughthe Internet or other network means, including various combinations ofall three. Accordingly, other embodiments are within the scope of thefollowing claims.

What is claimed is:
 1. A method, comprising: (a) receiving a firstrequest, wherein the request is received across a network by aprocessing device including a memory and a processor; (b) transmittinguser interface data, wherein the user interface data comprises: overheadmap data; and a click through link associated with a first real estateproperty indicated in the overhead map data, wherein the click throughlink initiates a second request for captured image data includingcomposite image data comprising at least a portion of multiple imageframes captured by a camera attached to a vehicle and calculated to beoriented at street level towards a centroid of a real estate parcelassociated with the first real estate property; (c) receiving the secondrequest; and (d) transmitting the captured image data and layer dataassociated with the first real estate property.
 2. The method of claim1, wherein the camera is a video camera, and wherein the captured imagedata includes multiple frames captured by the video camera.
 3. Themethod of claim 1, wherein the captured image data is captured by aplurality of cameras oriented at street level to the first real estateproperty, and wherein each of the plurality of cameras are approximatelyco-located.
 4. The method of claim 1, wherein the first request of (a)is a webpage request made by a browser.
 5. The method of claim 1,wherein the transmitting of (b) is performed in response to receivingthe first request in (a), wherein the overhead map data is used todisplay a map to a user, wherein the click through link is superimposedon the map and is visible to the user, and wherein the first real estateproperty is indicated on the map.
 6. The method of claim 1, wherein thesecond request of (c) is a webpage request made by a browser.
 7. Themethod of claim 1, wherein the transmitting of (d) is performed inresponse to receiving the second request in (c), wherein the layer datais superimposed on the captured image data, thereby generating acomposite image, and wherein the composite image is displayed to a user.8. The method of claim 1, wherein the layer data includes informationtaken from the group consisting of : geo-coded data, vector data,property county parcel data, landmark data or county property line data.9. The method of claim 1, wherein the captured image data associatedwith the first real estate property comprises an image that that iscentered on a centroid of a parcel on which the first real estateproperty is located.
 10. The method of claim 1, wherein the layer dataincludes information taken from the group consisting of : record data,sale price data of the first real estate property, sale price data of acomparable real estate property, owner data, or address data of thefirst real estate property.
 11. The method of claim 1, wherein the layerdata includes information taken from the group consisting of: MultipleListing Services (MLS) data or a visual cue data indicating where thefirst real estate property is located within the captured image data.12. The method of claim 1, wherein the camera is attached to a vehicletraveling at street level relative to the first real estate property,and wherein the captured image data comprises a plurality of imagescaptured from discrete points traversed by the vehicle while the vehicleis in proximity to the first real estate property.
 13. The method ofclaim 1, wherein the captured image data transmitted in (d) includes aplurality of images of the first real estate property.
 14. The method ofclaim 1, wherein the captured image data is captured via a plurality ofcameras, wherein each of the plurality of cameras are co-located, andwherein each of the plurality of cameras are oriented in a uniquedirection.
 15. The method of claim 14, wherein the captured image dataincludes three hundred and sixty degree street views about the pluralityof cameras.
 16. The method of claim 1, further comprising: (e)transmitting aerial data associated with the first real estate property,wherein the aerial data includes an overhead photo of the first realestate property.
 17. The method of claim 1, wherein the network in (a)is the Internet.
 18. The method of claim 1, further comprising: (e)receiving a third request for captured image data including an imagecaptured by the camera oriented at street level to a second real estateproperty; and (f) transmitting captured image data and layer dataassociated with the second real estate property.
 19. The method of claim18, wherein the third request of (e) is a webpage request made by abrowser, and wherein the network in (a) is the Internet.
 20. The methodof claim 18, wherein the transmitting of (f) is performed in response toreceiving the third request in (e), wherein the layer data issuperimposed on the captured image data including an image captured bythe camera oriented at street level to the second real estate property,thereby generating a composite image, and wherein the composite image isdisplayed to a user.